Field of the Invention
The present invention relates to a technique of producing surface normal information of a surface of an object.
Description of the Related Art
When acquiring an image by image capturing of an object through an image capturing apparatus such as a digital camera, a known method acquires not only luminance information, but also information of a shape of the object, in particular, information of a surface normal. Acquisition of a large amount of physical information relating to the object allows image production to be performed based on a physical model in image processing after the image capturing. For example, the image processing can produce an image in which an impression of the object is changed. The impression of the object is determined by factors such as shape information of the object, reflectance information thereof and light source information of a light source illuminating the object; as the shape information, it is effective to use surface normal information, not a three-dimensional shape. This is because physical behavior of light emitting from a light source and then being reflected by the object depends on a local surface normal.
A known method of acquiring the surface normal information acquires distance information by distance acquisition methods such as a triangulation method using laser light and a binocular stereo method and converts a three-dimensional shape calculated from the distance information into the surface normal information. However, these distance acquisition methods require a complicated apparatus, and an accuracy of the surface normal information thus obtained is insufficient. Moreover, methods using laser light such as the above distance acquisition methods cannot be applied to an object with luster and a transparent object.
Known methods of directly acquiring surface normal information include a photometric stereo method and a method of estimating the surface normal information from polarization information. The photometric stereo method (refer to Yasuyuki Matsushita, “Photometric Stereo”, Information processing society of Japan Research Report, Vol. 2011-CVIM-177, No. 29, pp. 1-12, 2011) assumes a reflection characteristic based on a surface normal of an object and a light source direction, and determines the surface normal from luminance information of the object acquired at multiple light source positions and the assumed reflection characteristic. The reflection characteristic of the object is often expressed by using a Lambert's reflection model according to a Lambert's cosine law.
In general, a reflected light from an object includes a specularly reflected light (also referred to as “a specularly reflected component”) and a diffusely reflected light (also referred to as “a diffusely reflected component”). The specularly reflected light is a light regularly reflected at an object surface and is a light Fresnel-reflected at the object surface (interface) according to Fresnel's equation. The diffusely reflected light is a light scattered back from inside the object after having transmitted through the object surface. The specularly reflected component cannot be expressed by the Lambert's cosine law, and the surface normal cannot be accurately calculated by the photometric stereo method when the specularly reflected component is included in the reflected light from the object observed by the image capturing apparatus.
Furthermore, shaded and shadow portions where any light from a light source does not reach generates a deviation from an assumed reflection model, and thereby the surface normal information of the object cannot be accurately acquired. Moreover, the diffusely reflected component from an object having a rough surface also deviates from the Lambert's cosine law. For example, Japanese Patent Laid-open No. 2010-122158 discloses a method of calculating a true surface normal from multiple surface normal candidates obtained by using four or more light sources. Japanese Patent No. 4435865 discloses a method of extracting a diffuse reflection area on a basis of a polarization state of a polarized light at an object surface when a polarization state of a light emitted from a light source or a position of the light source is changed and of applying the photometric stereo method to the extracted diffuse reflection area.
A known method of performing the estimation from the polarization information uses a polarization degree and a polarization direction as the polarization information. Japanese Patent No. 4435867 discloses a method of acquiring surface normal information of an object from multiple polarization images having different viewpoints. The method disclosed in Japanese Patent No. 4435867 estimates the surface normal information of the object by estimating one degree of freedom of a surface normal from the polarization direction for each viewpoint. The method acquires the surface normal information in an occlusion area where the information can be obtained only from one viewpoint, under a constraint condition that the area is an edge portion of the object.
However, the method disclosed in Japanese Patent Laid-open No. 2010-122158 cannot acquire the surface normal information of a specular object generating a small diffusely reflected component. When the object has shaded and shadow portions where any light from any light source does not reach and any diffusely reflected light under three or more light source conditions cannot be observed, the method acquires wrong surface normal information. However, constant acquisition of the diffusely reflected components under the three or more light sources for a common object requires image capturing with a larger number of light source conditions, which is difficult. Moreover, the method cannot estimate a correct surface normal when the diffusely reflected light does not obey the Lambert's cosine law.
In addition, the method disclosed in Japanese Patent No. 4435865 applies the photometric stereo method only in the diffuse reflection area, so that the method cannot acquire the surface normal information in a specular reflection area. The method also cannot estimate a correct surface normal when the diffusely reflected light does not obey the Lambert's cosine law.
Moreover, the method disclosed in Japanese Patent No. 4435867 cannot calculate the surface normal information in an area with a small polarization degree. Image capturing from a viewpoint where a large polarization degree is obtained requires a complicated image capturing apparatus for a common object, which is not desirable.